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Autofocus Methods 



This is a continuation of a previous Patent Application OB0405773.3, filed 15* March 2004. 



We now describe some specific control methods for autofocus cameras. For ease of 
explanation of operation of the control methods, we represent them schematically by way of 
flowcharts (Figs. 1 to 10). How the methods are implemented in an actual camera is a matter 
of engineering choice, and could be.for example, hard-wired control logic, a set of gates, 
register* and memory cells in a Field programmable logic array, or a general purpose DSP 
(digital signal processor) or microprocessor connected to input and output circuity executing 
a program that causes the ensemble to enact the control method(s). but other possibilities 
exist and are to be understood to be included in this invention. 

Again for ease of description we represent each of the autofocus control methods as a sub- 
method of operation of the entire camera system, autofocus itself simply being one of the 
many operations necessary for a camera to perform a useful function, the other operations 
possibly including, calibration of the sensitivity to light and appropriate closure of the 
aperture and/or change of exposure time, the "taking" of the picture <i.e. the final exposure to 
be captured and stored), and the transfer of the captured image from the image sensor of the 
camera to some storage medium. For this reason the flowcharts describing the autofocus 
control methods have START and RETURN control points. The START point is where the 
overall or general camera control system begins the autofocus process, and the RETURN 
point is where this iteration of the autofocus process is complete and general camera control 
method (hereinafter the General Camera Control Method or GCCM) continues before later 
re-entering the autofocus control method (hereinafter the Autofocus Control Method or 
ACM) again. 

Fig. 1 is a simplified generic ACM flowchart which underlies all of the more detailed 
methods described subsequently. To avoid repetitive description of operations, certain points 
in the control method are given labels of the form Labxxx where xxx is a string of one or more 
digits, and where the description says goto Labxxx it is to be understood that the operations at 
and after the label Labxxx are then to be carried out. 

Once the ACM is entered via the START box, the first operation is to initialize to suitable 
values the loop variables that are used to control the details of the rest of the method. The 
ACM also has a State variable that persists from one invocation of the method to the next, 
although (he ACM itself may change the value of the State variable. The essential point is that 
when the method returns control to the GCCM, the Suae is not lost, and unless deliberately 
changed by an external system e.g. the GCCM itself, the State variable will have the same 
value the next time the ACM is initiated. 

Next the State variable is checked and if the ACM is in Idling state (which means "do 
nothing") the ACM returns control directly to the GCCM. 

If instead the State is Initialisation state (which means, ,$ prepare for a new focus 
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measurement process cycle of operations"), then certain focus control variables are 
initialised to suitable values (including ih^ frames variable) after which: 
[LabOl] lh& frames variable is incremented. A test is then carried out to check if all 
the required frames have been done and: 

if so, the State is moved to the next state; i.e. the State variable value is 

changed; 

in either case the ACM then returns control to the OCCM and passes back a value 
currpos which describes the current focus-lens position. 

If instead the State is Running state, then a Figure of Merit (FOM) is determined for 
the image being received by the camera with the focus-lens at its current position 
(currpos). This newly derived FOM is then compared with any previous FOM values 
determined since the last Initialisation state: 

if the new FOM is the best so far, then it is remembered as a new value of best 

FOM so far, for future FOM comparisons; 
in either case, the ACM then works out how to alter the focal-lens position on the 
basis of this FOM measurement Then goto LabOl. 

If instead the State is Flyback* then the ACM causes the focus-lens to move to a 
known position. Then goto LabOL 

If instead the State is Track Focus then the frames variable is set to a large number, so 
that the rest of the algorithm will process many frames before it registers all required 
frames done. Then goto LabOl > 

Otherwise, if in none of the above states, set State to Idling and RETURN to the 
GCCM, 



The above generic description of the ACM does not describe in detail several crucial 
processes, which are now discussed in the more specific versions of the ACM illustrated by 
the flowcharts in Figs. 2 to 10. 

Big, 2 shows the first more specific implementation of the ACM described in general above. 
So in the first specific implementation a description of the loop variables and their initialised 
values can be seen. These include the number of initial steps, initSteps = 5 and flyback steps, 
flybackSieps = 5, the number of scan steps, scansteps = 25 , a zero value for the frames done 
variable/ra^s and an upper limit for the drive value, DrivMax « 255, which is specific to 
each and ©veiy implementation, and simply represents the maximum allowable value for the 
lens position drive signal* 

When the state on entry to the routine is initialisation, State = init, then specific values for the 
focus variables are defined as follows: best position bestPo® = 0, current position currpos = 0, 
beat Figure of Merit bestFOM = 0, and number of frames frames * tnitSieps. 

If in the running atate, State = rwmmg, then a current FOM valu© thkFOM m calculated by a 
call KO a aub-prosesg gmFQM (described below m two preferred fbrmatt, but My iuitabb 
method of determining an FOM cm be us©d here); this FOM ia than checked against 
current bm FOM, bastFOM* md if the nmv vnJue ia better, then hssiFQM h updated with 



Tmiy Hfcdley, 10/08/04 Vl.2 ' Q Lid 



\QVi33Z9 IQ-Aug-04 08:29 | 



t i i . : <. -ri +44 1223 422291 

lUG-2004 20S18 1 Linited "«■"* 

CONFIDENTIAL Autofocua Patent Addendum f «pg 3 g£ IS. 

value? of thisFOM, and the focus lens position for beat FOM, bestPos is updated from the 
current position currPos. Then a new lens position currPos to try is calculated, and the 
frames variable updated. 

If the system is in the flybaak state, State = Flyback, then the next lens position is B et to 0, 
currpos = 0, and ihc frames variable set to a suitable value for the flyback process, 
=flybacksteps in this version, after which the StateFrame variable is incremented (NB in the 
flowcharts the notation +-w is used to indicate that the variable * is incremented (by 1)). 

If the system is in the tracking state, State » track, then the frames variable is set to 100 in 
this version, before the Stateframe variable is incremented. 

These latter states than pass to a test to see if all required frames are done by comparing 
Stateframe wtth frames, and if so, the system state is modified according to its current state 
(init changes to running, running to flyback, flyback to Idle, Track to Track (Le. no change) 
and Idle to Idle (again, no change of state)). The last action is to move the lens to the newly 
computed position currpos by a call to the focal lens movement control routine Focus(). 

If the entry state was any other state, then the state is set to Idle before returning. 

Note that in the above description all the constant values to which the various variables are 
set to during the process are example values that have ben found to work well, but other 
values are also possible and useful in real implementations and these given here are merely 
guides to one particular implementation and are not meant to be limiting. 

Fig, 3 shows a second variant of the basic autofocus algorithm, very similar to that in Fig. 2, 
the principal Changes being: 

In the initialization phase ;■ 

initSteps = flybacksteps = 3 [instead of 5]; 
ScanSteps =15 [instead of 25] ; 
In the running state, the new position is calculated differently:- 
currpos = (StaKFrame+I)*Drivemax/scansteps' f 

Fig. 4 shows another slight variant of the method shown in Fig. 2, the principal differences 
being: 

In the running state, the new position is calculated differently:- 
currpos = StateFrame*Drivemax/scanstepsl 

After the test for all frames done (JStateframe>Jrames T) 

an extra test is inserted if the state is running, to see if the best position is 
greater than a certain large value (in this example, bestPas > 230?), and if the 
test succeeds then instead of moving to flyback state instead the system moves 
to idle state, after resetting the next position to the latest best-position, currpos 
= bestPos, just prior to the instruction to move to that new position. 

Alternatively, if the state is flyback, then state is reset to idle and again 
currpos is set to bestPos. 

Fig. 5 is much like Figs. 2, 3. and 4, with combinations from each, with one additional 

Fife: C.\D™nn»l. ScttoWAI, Dc Mtn «l Ww ..Cl«. * Or&^Foa^U*. *U1W n m chrt.Vl.il Otu.lu.wpa "~* 

Tony Hooley, 10/08/04 V1.2 1 L,a 



P - ©6. 



18-AUG-2904 2QI19 



1 Limit G-d 



HH4-4 1223 422291 



P 




CONHDEM1AL 



AutofbcuB Patent Addendum 



Page 4 of 18 



change: 



After all frames are done and the state is flyback: 

Instead of changing the state to idle, the state is left as flyback before returning 
control to the GCCM, 

Note that each of these variants shown in Figs. 2, 3, 4, S each have their own slightly different 
behaviour which are advantageous in certain situations and are all useful methods in their 
own right. 

Fig* 6 is yet another variant of autof ocus control method, with an element of dead-reckoning 
built into it, designed to reduce the effects of any hysteresis in the actuator that converts an 
electrical drive signal into a mechanical position for the focal-lens* Clearly hysteresis in such 
a component will cause the lens to move to different positions for the same drive signal due 
to the previous history of mechanical and drive states, so that depending on these histories, 
methods ignoring hysteresis will behave differently at different times, and in general less well 
than if hysteresis were absent The principal differences from the previous methods described 
are: 

In the tracking state;- 

the method uses the same control state sequence as the flyback state; 
after the test for all frames done has been satisfied, if the system is in the 
running state, then the next new position of lens, currpos is computed by use 
of a look up table (LIFT) using the stored bestPos value is the index into the 
table. This lookup table will contain pie-computed (static at design, assembly 
or test time, but possibly dynamically controlled values of position data, 
estimated to correct for the actual hysteresis of the actuator element). 



Fig. 7 shows a simplified method of continuous tracking position control, which continually 
attempts to bring the image into focus despite changes in camera position and changes of 
scene in the view of the camera. Its operation is as follows: 

Once started (START), a position step size S (which is the amount the lens drive signal 
will change during the next tracking iteration) is set to an initial value SO. 

An initial "current" direction of step (lens in or lens out) is chosen. 

[Supposition:] Next the lens is stepped by 5 in the current direction. 

At this point a focus Figure of Merit (focus quality) is measured, and if this FOM has 
mi changed by a large amount (what is considered a large amount is predetermined 
and stored as a constant as part of the method control), then the etep size S m 
imna&ed by an amount delta J \ but not allowed to exceed a maximum step size Smax 
whereupon control loops back to the StepPositim action. 



If th© FOM did chang© by a large amount and if th© fosus got woi'&e (i.e. FOM 
induced) th©n step siss ■§ ta increased by &n amow. rieiteS ugmn with a check 
to mmm. it mmmm feoumdfed by Smm b&fom looping bask to SttipPoststm 
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again. 

If the focus improved after the last big change in FOM. then the 
stepsize S is decreased by an amount delta3 t but constrained not to fall 
below a minimum value Smin. The step direction is then reversed, 
before looping back to StepPosition once more. 

The key points to note about this method are: 

a) There is no termination condition as such - the autofocus tracking process just 
continues ad infinitum until stopped by some external event - the subtle 
changes in step size give the process stability. This absence of termination 
condition prevents the method from falsely terminating on focus (FOM) 
plateaus or getting stuck in local minima, because inherent random noise in 
any real camera/imaging system will cause sufficient changes in the FOMs 
measured to allow the loop to escape from such conditions. 

b) The step size can increase (up to some maximum Smax) while the method is 
operating, Even if poor focus decisions have been made in the past, due maybe 
to an object passing into the field of view transiently, or to camera movement 
e.g. shake, or to the object of interest moving in and out of focus, or even due 
to some sort of error in the whole system, the gain of the system Ci.e, how 
much the focus position is shifted in response to a certain change in FOM) can 
still recover <i.e. increase) to bring the desired object back into focus in a 
reasonable time," and importantly to also help the system out of a transient false 
minima. 



c) The ability to detect a large change in focus quality allows the method to 
detect a scene change (i.e. field of view change). When this occurs the step 
size is increased to allow a focus position search for a new object in view, 
Even if the change of scene is not detected outright, the continuous running of 
the method with no tennination step enables a new object (position) to be 
brought into focus in a reasonable time. 

Some further important aspects of this last mentioned focus tracking method are: 

1) The largest allowable stepsize (Smax) should be constrained so that it is impossible to 
completely misB focussing on an object in the field of view by stepping over the lens 
position at which it is in focus, in one step. This maximum stepsize Smax can be 
calculated at system design time for any given sensor-chip+Jens+actuator+drive 
electronics combination. A useful and preferable value of Smax has been found to be 
around 1/6* of full-scale focal lens displacement. 

2) The minimum stepsize Smin is more difficult to pre-estimate at system design time, 
However, it is preferably smaller than half full-scale focal lens displacement. A useful 
and preferable value of Smin has been found to be around lW of full-scale focal 
lens displacement. 
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3) For an autofocus control system where the focal position drive signal is in the range 0 
to 255 (e.g. a digital system with a one-byte control word), the following values for 
each of the parameters of this control method are preferable (and for other overall 




initial stepsize 


SO 


= 48 


minimum stepsize 


Smin 


= 8 


maximum stepsize 


Smax 


= 48 


first stepfcizs increment 


delta! 


= 16 


second stepsize increment 


delta! 


= 6 


third stepsize increment 


delta3 


= 8 



Fig. 8 shows in more detail a practical implementation of a tracking focus method, which 
lacks some of the sophisticated details of the general method just outlined, but which is 
nonetheless useful in its own right. As shown here the autofocus process method appears to 
have simple termination points (the round-cornered terminal boxes at middle right and 
bottom left of Fig. 8 containing the word "return"), but in practice this is simply so that the 
external controlling process (the GCCM e.g.) may easily choose to terminate the otherwise 
looping process when required. So in practice the GCCM would call this process (shown in 
Fig. 8) repeatedly (as if the Fig. 8 process itself continuously looped between the entry box 
(round cornered box at top left containing "AF_Traok Slow(» and the previously mentioned 
exit terminal boxes), with some GCCM determined parameter causing termination, not the 
autofocusing process itself. 

As in previous methods described, the system has a .State parameter which largely determines 
the gross effect of an activation of the method, but a particular difference, due to the need 
during tracking of continuous looping through the method, is the very first condition test (first 
call?), which ensures that two parameters, currpos and Stepsize are initialised only on the 
. very first activation of the method, subsequent activations retaining the previous state or 
value of these. This method incorporates directly one or other of the previously described 
autofocus methods (as described with respect to Figs. 2, 3, 4, 5 or 6) which occurs after the 
conditional test (affjh defined? )=>YES in the process box containing {call the a/fjh fag. 
Algol/2/3, NoComp, deadreckj)> where the correspondence is between Atgo J and Fig. 2, 
Algol and Fig, 3, Algo3 and Fig. 4, NoComp and Fig. 5, or deadrsch and Fig. 6. The 
conditional test (a/f jh defined?) simply checks for the existence of a definition of which one 
of these functions to use, and if it doesn't exist, calls none of them. 

A new feature in the Fig, 8 method not seen in earlier versions described, is an FOM history 
memory, in this example implemented as an array of four FOM values FOMhist[0:3 7, 
together with an index histptr into this array which determines where the next value will be 
stored. During operation, to reduce the effects of noise and camera/scene movement, a 
number of (four in the example shown) successive FOMs are recorded and stored in the 
FOMhistU array, before $ny chung^s. to focal position via curvpos sr© mads. Thig makes the 
method eom^what slower than otherwise but the improvement in reliability and stability ii 
worthwhile in many situations. Thm choice of number of m&h averaging iterations may of 
courms h& vmi®d to mit my particular application* When* in thin axsanpl®, four FOMi ir© in 
the array, ®n awr&;p mrmvg over th© four values m computed, kisspzp m raast to zo>m» mid a 
diesk for reaching mffiiraam tern position (hitting "endsfcops**) is ©aimed out. The curcently 
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computed average FOM is compared with the previous average (if it exiBts, and if it doesn't 
then one is generated from the current value, stored as lastavg and used for the next iteration) 
and depending on whether the latest average FOM is better (more in focus) or woree than the 
last, and whether the lens was last moved in the positive or negative direction, the new 
position to be used, currpos, is produced either by incrementing or decrementing the current 
value by Stepsize, with the resulting value clipped to the allowable range (in this example 
between 0 and 255). The logic is clear from the flow diagram at Fig. 8. The method finishes 
by moving the lens to the new position currpos, before starting the next iteration if so allowed 
by the GCCM. 

Fig. 9 shows a more detailed implementation incorporating some of the features of the 
adaptive tracking process with dynamic changes to the stepsize as outlined above with respect 
to Fig. 7, and incorporating several of the features of the non-adaprive tracking method 
outlined with respect to Fig. 8, including the FOM history store and FOM averaging, and the 
first-call-only initialisation of certain parameters, and most particularly the parameters 
currpos, currstep (which replaces StepSize in this method), and stepdec. 

It will be seen that in several places in the method the size of step currstep to be next made in 
focal lens position, is altered as a function of the state of other parameters (NB the notation in 
several of the process boxes like those near bottom right of Fig, 9, viz. {currpos += currstep) 
and (currpos •^currstep') is standard C-language notation meaning that the variable currpos is 
to be incremented (+=) or decremented (-=) by the amount currstep). The value of currstep is 
also kept within bounds by the various checks against maxstep and minstep. 

As with the previously described autofocus methods; this method illustrated in Fig. 9 enlists a 
sub-process getFOMQ to determine a value for the Figure of Merit at the current position. 
Examples of such FOM determining methods will now be described. 

Fig. 10 shows a method (called here AFJ-OGFOMK) but which may be substituted for the < 
generic sub-method getFOMQ enlisted by the herein previously described autofocus methods) 
for determining a Figure of Merit. In this example the method works on an 8x8 (total block 
size ■ 64) block of image pixels which have been Discrete Cosine Transform (DCT) encoded 
(e.g. as may be found in common compressed image data formats). The two numeric blocks 
at top right of Fig. 10 represent the layout of a typical 8x8 block of elements in the complete 
set of DCT elements, with the first numeric block showing the index scheme used within the 
8x8 block (indexes running sequentially from 0 through 62, and the second numeric block 
showing the coefficients used at each index position, in the method. Note that in each block 
certain cells (15 in each) are greyed: the top set of greyed cells (i.e. in the top numeric block) 
are the indexes of those cells, the bottom set of greyed cells (i.e, in the lower numeric block) 
are the only non-zero coefficients used, keyed to the corresponding indexes in the top 
numeric block. Cells where the coefficients are zero are ignored and play no part in the 
method. 

In this example the total size of the image DCT values used is 176 x 144 pixels corresponding 
to a set of 22 x 18 » 396 blocks of 8x8=64 pixels each. These 396 blocks are processed in 
turn by the method. The list of 15 non-zero coefficients is loaded into array coefflist[], and 
the list of 15 indexes of the locations of the non-zero coefficients is loaded into indexhszfj. 
The FOM output value FOMout is initialised to zero as is the block index HMndx, and the 
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partial FOM value FOMpart, the latter within the outer loop of the method, On each trip 
round the outer loop the block index blkindx is incremented by the block size blksiz, and the 
output FOM FQMout incremented by half the value of the partial FOM FOMpart. The non- 
zero coefficient list index step is also reset to zero at each pass round the Outer loop. 

Within the inner loop each of the 15 nonzero coefficients coefflisl{0:14) is multiplied by its 
corresponding DCT value DCTb^blkmdx^indexlistfsiepJJ to give product tempres. where 
blkindx is an offset to the start of each 8x8 block of DCT values, and indexlistfstep] is the 
corresponding off&et within that block of the DCT value to be multiplied by the coefficient. 
The partial FOM value FQMpari is then incremented by the absolute value of tempres. 

There is also in the outer loop a block counter and when it determines that ail blocks within 
the DCT buffer DCTbufffJ have been processed, the method returns the FOM estimate 
FQMout. 



Other much simpler methods of determining a*n FOM may also be used as sub-method 
getFOMQ in the above autofocus methods. One-dimensional (ID) filters with as few as 9 taps 
work well, looking only at one-line of raw pixel data (i.e. not DCT values as used above) at a 
time. Alternatively two-dimensional filters looking at several lines of raw pixel data 
simultaneously will generally give improved performance over the ID methods, as instances 
of getFOMQ, but the improvements are not always necessary, quite reliable results being 
achievable with ID methods. 
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